MS&E 336 Lecture 15: Calibration

Calibration is a concept that tries to formalize a notion of quality for forecasters. For example, suppose a weatherman predicts each day whether the it will rain, or be sunny. Typically forecasters will predict such events in terms of probabilities, i.e., “There is a 30% chance of rain.” Given only the outcome that day, it is impossible to judge the quality of such a forecast. However, if we consider all days on which a forecaster said the probability of rain was x%, it is reasonable to expect that the fraction of such days on which it rained is exactly x%. This is precisely the notion of calibration. In this lecture we first define a notion of calibration that is appropriate for the study of games, and then prove that calibration is essentially a generalization of internal regret minimization. Formally, we will show that playing a best response to a calibrated forecast of the opponent is an internal regret minimizing strategy; and that using an internal regret minimizing algorithm, one can easily build a calibrated forecaster. Our presentation is based primarily on the corresponding paper of Foster and Vohra [2]. Throughout the lecture we consider a finite two-player game, where each player i has a finite pure action set Ai; let A = ∏